CompareDx: a Software Toolkit for Measuring the Performance of Programs that Generate Multiple Diagnoses

Introduction Evaluations of medical diagnosis programs have been carried out for several decades but for programs which produce multiple diagnoses there is a lack of suitable, well validated performance metrics. If a program reasons about only one (or a few) types of diagnosis, then the sensitivity and specificity of the program can readily be determined given a suitable standard diagnosis. However if the program is designed to reason about the possibility of dozens or hundreds of diagnoses other metrics may be required. Evaluating such programs usually requires a considerable amount of data per case and it is therefore difficult to collect more than 100 to 200 cases. This results in sparse data with many diagnoses appearing only once or twice in the evaluation (and many diagnoses not appearing at all). Calculating sensitivity and specificity for each diagnosis is therefore impractical, and only common diagnoses can be effectively evaluated. We have refined performance metrics for assessing diagnostic accuracy, evaluated them with data from a clinical evaluation study, and developed a Java program to implement the metrics efficiently.